1. Field of the Invention
The present invention relates to image processing apparatuses and image processing programs for processing images output from a single-chip image pickup system, a two-chip image pickup system, or a three-chip pixel-shifting image pickup system to generate a color digital image with three color component values at each pixel.
2. Description of the Related Art
Single-chip image pickup systems frequently used in, for example, digital cameras use a single-chip image pickup element having a different color filter for each pixel, and an image output from the image pickup element has only one type of color component value at each pixel. Thus, interpolation for compensating for missing color component values at each pixel is required to produce a color digital image. Interpolation for compensating for these missing color component values is also required in apparatuses which use a two-chip image pickup system or a three-chip pixel-shifting image pickup system.
This interpolation may cause the finally produced color images to undergo deterioration such as false color or blurring, unless ingenuity is exercised. For this reason, interpolation based on various methods has been proposed. Interpolation is classified roughly into two types. One is based on a method of edge detection and the other is based on a method of color correlation.
A technique based on edge detection is described in, for example, Japanese Unexamined Patent Application Publication No. 8-298669. The technique described in that Patent Application Publication will now be described with reference to FIGS. 32A and 32B. FIGS. 32A and 32B are illustrations for known interpolation based on edge detection.
It is assumed that a single-chip image pickup element is realized by, for example, a single-chip Bayer array image pickup element provided with color filters of a three-primary-color Bayer array, as shown in FIG. 32A. When neighborhood pixels of B5 in the form of a cross, which is a pixel of interest, are given, as shown in FIG. 32B, along the vertical direction and the horizontal direction, an interpolation value Gh for G in the horizontal direction and an interpolation value Gv for G in the vertical direction of the relevant pixel of interest are estimated as shown in Expression 1.Gh=(G4+G6)/2+(2×B5−B3−B7)/4  [Expression 1]Gv=(G2+G8)/2+(2×B5−B1−B9)/4
Then, evaluation values dH and dV are calculated as shown in Expression 2, where dH and dV indicate the direction in which more gaps exist, i.e., the horizontal direction or the vertical direction.dH=|G4−G6|+|B3−2×B5+B7|  [Expression 2]dV=|G2−G8|+|B1−2×B5+B9|Symbol |x| represents the absolute value of x. The interpolation value Gh or Gv is used depending on the direction corresponding to the evaluation value dH or dV calculated in this manner, whichever is determined to be smaller or flatter.
On the other hand, a technique based on color correlation is described in, for example, Japanese Unexamined Patent Application Publication No. 11-215512. The technique described in this Patent Application Publication will now be described with reference to FIG. 33. FIG. 33 is an illustration for known interpolation based on color correlation.
As shown in FIG. 33, it is assumed that, in the vicinity of a pixel of interest X, a linear correlation as represented by Expression 3 using factors αqp and βqp is established between Vp and Vq, which are respectively the component values of any two color component types p and q selected from among the color components of the color filters included in the image pickup element (e.g., r, g, and b for color filters on a primary-color system).Vq=αqp×Vp+βqp  [Expression 3]Here, p and q are one of r, g, and b, if the color filters included in the image pickup element are based on, for example, the primary-color system, and the component values Vp and Vq corresponding to the color components p and q, respectively, are values of pixels in a neighborhood of the pixel of interest X.
Then, the pixels in a neighborhood U determined in the vicinity of the pixel of interest X are classified into three subsets Ur, Ug, and Ub, according to the obtained color component type r, g, or b. The factors αqp and βqp are estimated from Expression 4 shown below based on the average Ac (where, c is one of r, g, and b) and the standard deviation Sc (where, c is one of r, g, and b) of the pixel values in each of the subsets.αqp=Sq/Sp, βqp=Aq−αqpAp  [Expression 4]
This technique is based on the concept that the ratio of the width of the distribution of the color component q (standard deviation Sq) to the width of the distribution of color component p (standard deviation Sp) gives the gradient αqp of the linear correlation shown in Expression 3 and that, furthermore,.a straight line with the obtained gradient αqp passes through a point plotted by the mean value Ap of the color component p and the mean value Aq of the color component q.
Then, based on the estimated factors αqp and βqp, the value Xm of a missing color component m at the pixel of interest X is calculated from the value Xe of the color component e obtained at the pixel of interest X, as shown in Expression 5 below.Xm=αme×Xe+βme  [Expression 5]
Here, m and e are each one of r, g, and b.
As described above, the known technique based on color correlation is characterized in that the correlation between pixel values of different color components in the neighborhood of the pixel of interest can be estimated.
On the other hand, in an actual image pickup system, variance in sensitivity may occur between even pixels that use the same color filters and in fact should produce the same color components, due to a difference in gain between lines or slightly different spectral characteristics of the used color filters.
In all the above-described techniques, the color component obtained at the pixel of interest is used without any change for the interpolation result. Hence, when a missing color component at the pixel of interest is to be calculated, information regarding the pixel of interest itself is reflected in the result. In general, it is more efficient to use, for the interpolation result, information regarding the original color component obtained at each pixel without any change. If the sensitivity differs at pixels where the same color components are obtained, however, this processing causes the sensitivity variance between the pixels to be directly reflected in the interpolation result. As a result, a side effect, such as the appearance of a slight grid pattern, occurs.